A Review of Emotion Recognition Based on EEG Signals

Introduction

Emotions can reflect people’s daily psychological activities [1] and play an important role in daily communication. The positive and negative emotions affect human health and working state [2]. Emotion recognition refers to identifying the corresponding emotional state through human actions or physiological responses. For emotion research, only by quickly and accurately judging people’s emotional state, can a more natural and better humancomputer interaction be achieved [3]. For example, in daily driving, real-time monitoring of the driver’s emotional state through the intelligent system, judging the driver’s psychological state and giving necessary reminders can reduce unnecessary accidents. In criminal interrogation, by monitoring the emotional state of the interrogated person during the interrogation, we can judge the authenticity of the confession and optimize the coping strategy.

In the existing research, physical signals such as facial expression, body posture and sound are easy to collect and have a certain research basis, which is very popular in the early stage of the research. However, the disadvantages are also obvious. This kind of physical signal is affected by the subject’s subjective consciousness, and its reliability is poor [4]. Therefore, the research on physiological signals such as EEG, skin temperature and heart rate has gradually become the mainstream. Such signals are difficult to camouflage during acquisition, and more reliable results can be obtained [5]. Among them, the research on emotion recognition using EEG signals has a high accuracy, so it has become the focus of research.

Emotional Representation Model

Discrete emotion model and dimensional emotion model are commonly used emotion quantification models [6]. Discrete emotion model indicates that emotion is composed of several discrete and independent psychological feelings. The most recognized is Ekman’s six basic emotions, including happiness, sadness, anger, fear, surprise, and disgust [7]. Other emotions can be combined from these six basic emotions. Discrete emotion model can use simple words to describe emotion, but it cannot describe the process of emotion generation, development, and disappearance.

In order to better study emotion, researchers established a dimensional emotion model. Lang PJ [8] proposed a valencearousal model, which maps emotions into this two-dimensional space. Valence (also called pleasure) represents positive (pleasant) and negative (unpleasant) situations, and arousal represents the intensity of human emotions. This model will divide some emotions into the same area, which is difficult to distinguish. Therefore Albert Mehrabian [9] proposed the Pleasure-Arousal-Dominance (PAD) model, adding the dominance dimension to the valence-arousal model, which represents human control under certain emotions.

EEG Signal Characteristics

EEG signal is a weak physiological signal, which is easily interfered by the outside world in the process of acquisition [10]. The common interferences include eye movement interference, EMG interference, ECG interference, high-frequency noise interference and so on. These interferences will not only affect the acquisition of EEG signals, but also increase the difficulty of EEG signal analysis. In order to facilitate subsequent research, regular signals are often obtained through preprocessing. In the field of emotion recognition, the features of EEG signals are mainly divided into time domain features, frequency domain features, time-frequency domain features, and spatial domain features. This section will introduce the common EEG features from the above four aspects.

Time Domain Features

EEG signals are mostly collected in the form of time domain, and the time domain feature is the statistical feature of the signal. [11] used three domain methods to calculate the characteristics of the time series of any sample data of EEG and achieved good classification results. [12] designed five domain features, including standard deviation, first-order difference means absolute value and second-order difference mean absolute value, etc., and fused them to improve the performance of emotion recognition.

Frequency Domain Features

The research on frequency domain features is to show the frequency information of the signal, which is mainly to convert the original time domain signal to frequency domain through Fourier transform to obtain the power characteristics of different frequency bands. Frequency domain analysis usually divides EEG signals into delta frequency band (0.5-4Hz), theta frequency band (4-8Hz), alpha frequency band (8-13Hz), beta frequency band (13- 30Hz) and gamma frequency band (30-50Hz). Li D H [13] used the short-time Fourier transform to convert the time domain signal into the frequency domain, respectively calculated the power spectral density features of Theta, Alpha, Beta and Gamma frequency bands and fused them with the facial expression features. The Long Short- Term Memory (LSTM) network is used for emotion recognition, and a good recognition effect is obtained.

Time-Frequency Domain Features

To better study the EEG signal, the time domain signal and the frequency domain signal are combined to study the change of the signal frequency with time. Time-frequency analysis can more comprehensively reflect the characteristic information of EEG signals. Wavelet transform decomposes the low-frequency part of the signal [14], and there are two basic types: continuous wavelet transform (CWT) and discrete wavelet transform (DWT). Hilbert- Huang transform is composed of empirical mode decomposition and Hilbert spectral analysis. Compared with traditional methods, it has more advantages in dealing with nonlinear and nonstationary signals. 15. Ning Zhuang [15] decomposed the IMF component by EMD method and used the three features of IMF1 as the features of emotion recognition.

Spatial Domain Features

Spatial analysis is a spatial distribution pattern composed of electrical levels placed in different positions on the cerebral cortex during EEG signal acquisition. Different brain regions also respond differently to emotion. [16] considered the complementarity between temporal, spatial and spectral features. A parallel spatialtemporal stream and spatial-spectral stream are designed to capture spatial-temporal and spatial-spectral features.

EEG Emotion Recognition Dataset

The commonly used data sets in EEG emotion recognition are introduced as follows:.

a) DEAP:This dataset was created by Koelstra et al. [17] to analyze human emotional states. It recorded EEG signals and peripheral physiological signals of 32 subjects while watching music videos, as well as frontal videos of 22 of them. The subjects were asked to give self-emotional assessments ranging from 1 to 9 from the dimensions of arousal, valence and dominance.

b) SEED [18]:The dataset, released by Shanghai Jiao Tong University, contains EEG signals collected while subjects watched movie clips. By watching about 4 minutes of videos containing positive, negative and neutral emotions, the EEG signals of 62 channels of 15 subjects were recorded.

c) SEED-IV [19]:In this dataset, 168 movie clips were selected as the material library of four emotions: happiness, sadness, fear and neutrality. The EEG signals of 44 subjects and the evaluation of arousal dimension and valence dimension were recorded.

d) Dreamer [20]:The dataset, released by the University of the West of Scotland, has 23 subjects watching 18 movie clips. The subjects’ EEG and ECG signals were recorded under the audio-visual stimulation, and their emotional state w5as selfassessed from the arousal dimension, valence dimension and dominance dimension.

e) CAS-THU [21]:This dataset was jointly proposed by the Institute of Psychology of the Chinese Academy of Sciences and Tsinghua University. It used 16 videos to induce 8 discrete emotions, including joy, amusement, tenderness, disgust, fear, anger, sadness and neutrality. EEG signals of 30 subjects were collected.

Summary and Prospect

Emotion recognition is very important for the research of human-computer interaction. It is widely used in disease treatment, brain computer interface, education and learning. This paper introduces the research of emotion recognition based on EEG from three aspects: emotion model, EEG characteristics and EEG emotion dataset.

The research of EEG signal has achieved good development. To improve the effect of classification task, it is more important to select appropriate single or fused physiological signals. However, there are few studies using cross subjects and fusion features, so it is necessary to increase and standardize the collection of data sets to facilitate researchers to conduct more in-depth research. It is believed that with the continuous development of brain science research, timely and rapid analysis of human emotions through EEG signals will become a reality.

Acknowledgements

This paper is supported by the national key research and development program (NO. 2020YFC0811004), high level talent scientific research start-up fund (NO.107051360021XN090/001), and 2019 Beijing basic scientific research business innovation team project (NO. 110052971921/003).

References

• Huang XT (1991) Introduction to Psychology. Beijing: Peoples Education Press.
• Luo JY (2020) Review of EEG-Based Emotion Recognition. Modern Computer 08: 68-71.
• Lu B L, Zhang Y Q, Zheng WL (2021) A survey of affective brain-computer interface. Chinese Journal of Intelligent Science and Technology 3(01): 36-48.
• Zhao GZ, Song JJ, Ge Y (2016) Advances in emotion recognition base-d on physiological big data. Computer Research and Development 53(1): 80-92.
• Rana Seif Fathalla, Wafa Saad Alshehri (2020) Emotions Recognition and Signal Classification: A State-of-the-Art. International J Synthetic Emotions (IJSE) 11(1).
• Zhang G H, Yu M J, Chen G (2019) A review of EEG features for emotion recognition. Sci Sin Inform 49: 1097-1118.
• Ekman P, Frieson WV (1971) Constants Across Cultures In The Face And Emotion. J Pers Soc Psychol 17(2): 124-133.
• Lang P J (1995) The emotion probe. Studies of motivation and attention. Am psychol 50(5).
• Albert Mehrabian (1997) Comparison of the PAD and PANAS as models for describing emotions and for differentiating anxiety from depression. J Psychopathol Behav Assessment 19(4): 331-357.
• Shu L, Xie JY, Yang MY (2018) A Review of Emotion Recognition Using Physiological Signals. Sensors 18(7): 2074.
• Li X Z, Bao W, Xie N G (2022) Emotion recognition based on electroencephalography signals. Beijing Biomedical Engineering 41(01): 8-16+48.
• Zhang B, Liu L, Yang L B (2021) Research on Emotion Classification of EEG Features Based on Time and Frequency Domain.
• J Changchun University of Sci Technol 44(05): 51-57.
• Li D H, Wang Z, Wang C H (2019) The Fusion of Electroencephalography and Facial Expression for Continuous Emotion Recognition.
• Deng L N, Wang X L (2021) A Review of Emotion Recognition Based on Physiological Signals. Internet of Things Technologies 11(7): 33-41.
• Ning Zhuang, Ying Zeng, Li Tong, Chi Zhang, Hanming Zhang, et al. (2017) Emotion Recognition from EEG Signals Using Multidimensional Information in EMD Domain. BioMed res int.
• Jia Z, Lin Y, Cai X (2020) SST-EmotionNet: Spatial-Spectral-Temporal based Attention 3D Dense Network for EEG Emotion Recognition MM '20: The 28th ACM International Conference on Multimedia. ACM.
• S Koelstra (2012) DEAP: A Database for Emotion Analysis; Using Physiological Signals. IEEE Transactions on Affective Computing 3(1): 18-31.
• Zheng W L, Lu B L (2015) Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks. IEEE Transactions on Autonomous Mental Development 7(3): 1.
• Zheng W L, Lu B L (2015) Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks. IEEE Transactions on Autonomous Mental Development 7(3): 1.
• Katsigiannis S, Ramzan N (2017) DREAMER: A Database for Emotion Recognition Through EEG and ECG Signals from Wireless Low-cost Off-the-Shelf Devices. IEEE J Biomed Health Informatics :1.
• Liu Y J, Yu M J, Zhao G Z (2018) Real-Time Movie-Induced Discrete Emotion Recognition from EEG Signals. IEEE Transactions on Affective Computing 9(4).